{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "celltoolbar": "Slideshow",
    "colab": {
      "name": "interspeech2019_tts.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "rise": {
      "scroll": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2nNdAQGLqYEq",
        "colab_type": "text"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/interspeech2019_tts.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wA10eXyHIv8X"
      },
      "source": [
        "# [T6] Advanced methods for neural end-to-end speech processing - unification, integration, and implementation -\n",
        "\n",
        "### Part4 : Building End-to-End TTS System\n",
        "\n",
        "Speaker: [**Tomoki Hayashi**](https://github.com/kan-bayashi)\n",
        "\n",
        "Department of informatics, Nagoya University  \n",
        "Human Dataware Lab. Co., Ltd.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vduenPiiqYEr",
        "colab_type": "text"
      },
      "source": [
        "## Google colaboratory\n",
        "\n",
        "- Online Jupyter notebook environment\n",
        "    - Can run python codes\n",
        "    - Can also run linux command with ! mark\n",
        "    - Can use a signal GPU (K80)\n",
        "- What you need to use\n",
        "    - Internet connection\n",
        "    - Google account\n",
        "    - Chrome browser (recommended)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d3aeAtC5qYEr",
        "colab_type": "text"
      },
      "source": [
        "## Usage of Google colaboratory\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/colab_usage.png?raw=1 width=80%>\n",
        "</div>\n",
        "\n",
        "- Do not close the browser\n",
        "- Do not sleep your laptop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gEzrjMKyqYEs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# example of the commands\n",
        "print(\"hello, world.\")\n",
        "!echo \"hello, world\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5cxwcM3MqYEu",
        "colab_type": "text"
      },
      "source": [
        "## TOC\n",
        "\n",
        "0. Installation\n",
        "1. Introduction of ESPnet TTS\n",
        "2. Demonstration of the ESPnet TTS recipe\n",
        "3. Demonstration of the use of TTS pretrained models\n",
        "4. Demonstration of the use of ASR pretrained models\n",
        "5. Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8xWr9FADn0Rf"
      },
      "source": [
        "## 0. Installation\n",
        " \n",
        "It takes around 3 minutes. Please keep waiting for a while.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "J6k0hPgzjxA5",
        "colab": {}
      },
      "source": [
        "# OS setup\n",
        "!cat /etc/os-release\n",
        "!apt-get install -qq bc tree sox\n",
        "\n",
        "# espnet setup\n",
        "!git clone --depth 5 https://github.com/espnet/espnet\n",
        "!pip install -q torch==1.1\n",
        "!cd espnet; pip install -q -e .\n",
        "\n",
        "# download pre-compiled warp-ctc and kaldi tools\n",
        "!espnet/utils/download_from_google_drive.sh \\\n",
        "    \"https://drive.google.com/open?id=13Y4tSygc8WtqzvAVGK_vRV9GlV7TRC0w\" espnet/tools tar.gz > /dev/null\n",
        "!cd espnet/tools/warp-ctc/pytorch_binding && \\\n",
        "    pip install -U dist/warpctc_pytorch-0.1.1-cp36-cp36m-linux_x86_64.whl\n",
        "\n",
        "# make dummy activate\n",
        "!mkdir -p espnet/tools/venv/bin && touch espnet/tools/venv/bin/activate\n",
        "!echo \"setup done.\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0K8skzW-qYEx",
        "colab_type": "text"
      },
      "source": [
        "## 1. Introduction of ESPnet TTS\n",
        "\n",
        "- Follow the [Kaldi](https://github.com/kaldi-asr/kaldi) style recipe\n",
        "- Support three E2E-TTS models and their variants\n",
        "- Support four corpus including English, Japanese, Italy, Spanish, and Germany\n",
        "- Support pretrained WaveNet-vocoder (Softmax and MoL version)\n",
        "\n",
        "Samples are available in https://espnet.github.io/espnet-tts-sample/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GJtoIfEXqYEy",
        "colab_type": "text"
      },
      "source": [
        "### Supported E2E-TTS models\n",
        "\n",
        "- [**Tacotron 2**](https://arxiv.org/abs/1712.05884): Standard Tacontron 2\n",
        "- [**Multi-speaker Tacotron2**](https://arxiv.org/pdf/1806.04558.pdf): Pretrained speaker embedding (X-vector) + Tacotron 2\n",
        "- [**Transformer**](https://arxiv.org/pdf/1809.08895.pdf): TTS-Transformer\n",
        "- [**Multi-speaker Transformer**](): Pretrained speaker embedding (X-vector) + TTS-Transformer\n",
        "- [**FastSpeech**](https://arxiv.org/pdf/1905.09263.pdf): Feed-forward TTS-Transformer\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZnZflXs2qYEy",
        "colab_type": "text"
      },
      "source": [
        "### Other remarkable functions\n",
        "\n",
        "- [**CBHG** (Convolutional Bank Highway network Gated recurrent unit)](https://arxiv.org/pdf/1703.10135.pdf): Network to convert Mel-filter bank to linear spectrogram\n",
        "- [**Forward attention**](https://arxiv.org/pdf/1807.06736.pdf): Attention mechanism with causal regularization\n",
        "- [**Guided attention loss**](https://arxiv.org/pdf/1710.08969.pdf): Loss function to force attention to be diagonal"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cI1PdlrsqYEz",
        "colab_type": "text"
      },
      "source": [
        "### Supported corpora\n",
        "\n",
        "- [`egs/jsut/tts1`](https://sites.google.com/site/shinnosuketakamichi/publication/jsut): Japanese single female speaker. (48 kHz, ~10 hours)\n",
        "- [`egs/libritts/tts1`](http://www.openslr.org/60/): English multi speakers (24 kHz, ~500 hours).\n",
        "- [`egs/ljspeech/tts1`](https://keithito.com/LJ-Speech-Dataset/): English single female speaker (22.05 kHz, ~24 hours).\n",
        "- [`egs/m_ailabs/tts1`](https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/): Various language speakers (16 kHz, 16~48 hours)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QhW6ey37Kj7y"
      },
      "source": [
        "## 2. Demonstration of ESPnet-TTS recipes\n",
        " \n",
        "Here use the recipe `egs/an4/tts1` as an example.  \n",
        "\n",
        "Unfortunately, `egs/an4/tts1` is too small to train,\n",
        "but the flow itself is the same as the other recipes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kK-qFOFupi-I"
      },
      "source": [
        "Always we organize each recipe placed in `egs/xxx/tts1` in the same manner:\n",
        "\n",
        "- `run.sh`: Main script of the recipe.\n",
        "- `cmd.sh`: Command configuration script to control how-to-run each job.\n",
        "- `path.sh`: Path configuration script. Basically, we do not need to touch.\n",
        "- `conf/`: Directory containing configuration files e.g.g.\n",
        "- `local/`: Directory containing the recipe-specific scripts e.g. data preparation.\n",
        "- `steps/` and `utils/`: Directory containing kaldi tools."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "X6T2rNlSpVMV",
        "colab": {}
      },
      "source": [
        "# move on the recipe directory\n",
        "import os\n",
        "os.chdir(\"espnet/egs/an4/tts1\")\n",
        "\n",
        "# check files\n",
        "!tree -L 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aGX_y4RMpqIK"
      },
      "source": [
        "Main script `run.sh` consists of several stages:\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_brief_overview.png?raw=1 width=75%>\n",
        "</div>\n",
        "\n",
        "- **stage -1**: Download data if the data is available online.\n",
        "- **stage 0**: Prepare data to make kaldi-stype data directory.\n",
        "- **stage 1**: Extract feature vector, calculate statistics, and normalize.\n",
        "- **stage 2**: Prepare a dictionary and make json files for training.\n",
        "- **stage 3**: Train the E2E-TTS network.\n",
        "- **stage 4**: Decode mel-spectrogram using the trained network.\n",
        "- **stage 5**: Generate a waveform using Griffin-Lim.\n",
        "\n",
        "From **stage -1 to 2 are the same as the ASR** recipe."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q8HvbWyiqYE3",
        "colab_type": "text"
      },
      "source": [
        "### Detail overview\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_overview.png?raw=1 width=90%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l6Zhc4iKLKeC"
      },
      "source": [
        "### Stage -1: Data download\n",
        "\n",
        "This stage downloads corpus if it is available online.\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_stage-1.png?raw=1 width=80%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8CNFWXETAeYC",
        "colab": {}
      },
      "source": [
        "# run stage -1 and then stop\n",
        "!./run.sh --stage -1 --stop_stage -1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lzqGzOqpw741"
      },
      "source": [
        "`downloads` directory is created, which containing downloaded an4 dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Y3FTgWO7w0Rf",
        "colab": {}
      },
      "source": [
        "!tree -L 2 downloads"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wa1OkxN3xSRx"
      },
      "source": [
        "### Stage 0: Data preparation\n",
        "\n",
        "This stage prepares kaldi-style data directories.\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_stage0.png?raw=1 width=80%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ozLaRF1LA3Aq",
        "colab": {}
      },
      "source": [
        "# run stage 0 and then stop\n",
        "!./run.sh --stage 0 --stop_stage 0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bF6wu5n3x9gD"
      },
      "source": [
        "Two kaldi-style data directories are created:  \n",
        "- `data/train`: data directory of training set\n",
        "- `data/test`: data directory of evaluation set  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5Ch0fonjxep5",
        "colab": {}
      },
      "source": [
        "!tree -L 2 data"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EwPydMzvxbcA"
      },
      "source": [
        "`wav.scp`: \n",
        "- Each line has `<utt_id> <wavfile_path or command pipe>`\n",
        "- `<utt_id>` must be unique\n",
        "\n",
        "`text`:\n",
        "- Each line has `<utt_id> <transcription>`\n",
        "- Assume that `<transcription>` is cleaned\n",
        "\n",
        "`utt2spk`:\n",
        "- Each line has `<utt_id> <speaker_id>`\n",
        "\n",
        "`spk2utt`:\n",
        "- Each line has `<speaker_id> <utt_id> ... <utt_id> `\n",
        "- Can be automatically created from `utt2spk` \n",
        "\n",
        "In the ESPnet, speaker information is **not used for any processing.**   \n",
        "Therefore, **`utt2spk` and `spk2utt` can be a dummy.**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "S5PJH9zxyqNg",
        "colab": {}
      },
      "source": [
        "!head -n 3 data/train/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XEgfec6u1KWA"
      },
      "source": [
        "### Stage 1: Feature extration\n",
        "\n",
        "This stage performs feature extraction, statistics calculation and normalization.\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_stage1.png?raw=1 width=80%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NfDgtBtdqYFE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# hyperparameters related to stage 1\n",
        "!head -n 28 run.sh | tail -n 8"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "znHW0IwbBX0o",
        "colab": {}
      },
      "source": [
        "# run stage 1 with default settings\n",
        "!./run.sh --stage 1 --stop_stage 1 --nj 4"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "e9OS6JIp474Z"
      },
      "source": [
        "Raw filterbanks are saved in `fbank/` directory with `ark/scp` format. \n",
        "\n",
        "- `.ark`: binary file of feature vector\n",
        "- `.scp`: list of the correspondance b/w `<utt_id>` and `<path_in_ark>`.  \n",
        "\n",
        "Since feature extraction can be performed for split small sets in parallel, raw_fbank is split into `raw_fbank_*.{1..4}.{scp,ark}`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PPI8NLsGqYFJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!tree -L 2 fbank"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tuPfH0ly46Ko",
        "colab": {}
      },
      "source": [
        "!head -n 3 fbank/raw_fbank_train.1.scp"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oj1bx25q5bl_"
      },
      "source": [
        "These files can be loaded in python via a great tool **kaldiio** as follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "3K0uiQgB5aYK",
        "colab": {}
      },
      "source": [
        "import kaldiio\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# load scp file\n",
        "scp_dict = kaldiio.load_scp(\"fbank/raw_fbank_train.1.scp\")\n",
        "for key in scp_dict:\n",
        "    plt.imshow(scp_dict[key].T[::-1])\n",
        "    plt.title(key)\n",
        "    plt.colorbar()\n",
        "    plt.show()\n",
        "    break\n",
        "    \n",
        "# load ark file\n",
        "ark_generator = kaldiio.load_ark(\"fbank/raw_fbank_train.1.ark\")\n",
        "for key, array in ark_generator:\n",
        "    plt.imshow(array.T[::-1])\n",
        "    plt.title(key)\n",
        "    plt.colorbar()\n",
        "    plt.show()\n",
        "    break"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-Au70T7i8ctz"
      },
      "source": [
        "Some files are added in `data/train`:\n",
        "- `feats.scp`: concatenated scp file of `fbank/raw_fbank_train.{1..4}.scp`.  \n",
        "- `utt2num_frames`: Each line has `<utt_id> <number_of_frames>` ."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ySykrx407Tek",
        "colab": {}
      },
      "source": [
        "!tree data/train"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SpbfzwsPqYFR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!head -n 3 data/train/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qt4s7lkl9miZ"
      },
      "source": [
        "`data/train/` directory is split into two directories:\n",
        "- `data/train_nodev/`: data directory for training\n",
        "- `data/train_dev/`: data directory for validation\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Yq6WNznw9P4z",
        "colab": {}
      },
      "source": [
        "!tree data/train_*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Fw_KBzQCBN0E"
      },
      "source": [
        "`cmvn.ark` is saved in `data/train_nodev`, which is the statistics file.  \n",
        "(cepstral mean variance normalization: `cmvn`)  \n",
        "This file also can be loaded in python via kaldiio."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h_7K3wBEqYFV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!tree data/train_nodev"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pTCG1CPRDN5H"
      },
      "source": [
        "Normalized features for train, dev, and eval sets are dumped in\n",
        "- `dump/{train_nodev,train_dev,test}/*.{ark,scp}`.  \n",
        "\n",
        "These ark and scp can be loaded as the same as the above procedure."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JazwdIVbDNFs",
        "colab": {}
      },
      "source": [
        "!tree dump/*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eYtmQCDdDk0d"
      },
      "source": [
        "### Stage 2: Dictionary and json preparation\n",
        "\n",
        "This stage creates char dict and integrate files into a single json file.\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_stage2.png?raw=1 width=80%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "fgWDdveUBctQ",
        "colab": {}
      },
      "source": [
        "# run stage 2 and then stop\n",
        "!./run.sh --stage 2 --stop_stage 2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IB-eU-W1FNLk"
      },
      "source": [
        "- Dictionary file is created in `data/lang_1char/`.  \n",
        "- Dictionary file consists of `<token>` `<token index>`.  \n",
        "    - `<token index>` starts from 1 because 0 is used as padding index.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "q8S74SjdFXcF",
        "colab": {}
      },
      "source": [
        "!tree data/lang_1char"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7FwqZAD_qYFc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cat data/lang_1char/train_nodev_units.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "b-QO0MLGGaBO"
      },
      "source": [
        "Three json files are created for train, dev, and eval sets as \n",
        "- `dump/{train_nodev,train_dev,test}/data.json`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "BrYKpYVUGh6K",
        "colab": {}
      },
      "source": [
        "!tree dump -L 2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CoRMOHDnG1Xk"
      },
      "source": [
        "Each json file contains all of the information in the data directory.\n",
        "\n",
        "- `shape`: Shape of the input or output sequence.\n",
        "- `text`: Original transcription.\n",
        "- `token`: Token sequence of the transcription.\n",
        "- `tokenid` Token id sequence converted with `dict` of the transcription"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "O_My36NFHAVI",
        "colab": {}
      },
      "source": [
        "!head -n 27 dump/train_nodev/data.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "O88CpXHtHnZA"
      },
      "source": [
        "Now ready to start training!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QUkLphZbIw51"
      },
      "source": [
        "### Stage 3: Network training\n",
        "\n",
        "This stage trains E2E-TTS network.\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_stage3.png?raw=1 width=80%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2qUlnk5DqYFi",
        "colab_type": "text"
      },
      "source": [
        "Training setting can be specified by `train_config`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "b-f5K2bYOaVY",
        "colab": {}
      },
      "source": [
        "# check hyperparmeters in run.sh\n",
        "!head -n 31 run.sh | tail -n 2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dtnOhdUxqYFk",
        "colab_type": "text"
      },
      "source": [
        "Training configuration is written in `.yaml` format file.  \n",
        "Let us check the default configuration `conf/train_pytroch_tacotron2.yaml`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3JXP8AmZqYFk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cat conf/train_pytorch_tacotron2.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6_QmCAS1OlD2"
      },
      "source": [
        "Let's change the hyperparameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "f2Q9sYVxOjzm",
        "colab": {}
      },
      "source": [
        "# load configuration yaml\n",
        "import yaml\n",
        "with open(\"conf/train_pytorch_tacotron2.yaml\") as f:\n",
        "    params = yaml.load(f, Loader=yaml.Loader)\n",
        "\n",
        "# change hyperparameters by yourself!\n",
        "params.update({\n",
        "    \"embed-dim\": 16,\n",
        "    \"elayers\": 1, \n",
        "    \"eunits\": 16,\n",
        "    \"econv-layers\": 1,\n",
        "    \"econv-chans\": 16,\n",
        "    \"econv-filts\": 5,\n",
        "    \"dlayers\": 1,\n",
        "    \"dunits\": 16,\n",
        "    \"prenet-layers\": 1,\n",
        "    \"prenet-units\": 16,\n",
        "    \"postnet-layers\": 1,\n",
        "    \"postnet-chans\": 16,\n",
        "    \"postnet-filts\": 5,\n",
        "    \"adim\": 16,\n",
        "    \"aconv-chans\": 16,\n",
        "    \"aconv-filts\": 5,\n",
        "    \"reduction-factor\": 5,\n",
        "    \"batch-size\": 128,\n",
        "    \"epochs\": 5,\n",
        "    \"report-interval-iters\": 10,\n",
        "})\n",
        "\n",
        "# save\n",
        "with open(\"conf/train_pytorch_tacotron2_mini.yaml\", \"w\") as f:\n",
        "    yaml.dump(params, f, Dumper=yaml.Dumper)\n",
        "\n",
        "# check modified version\n",
        "!cat conf/train_pytorch_tacotron2_mini.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IKNbf7YkqYFo",
        "colab_type": "text"
      },
      "source": [
        "Also, we provide `transformer` and `fastspeech` configs.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O-SWxC2EqYFp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cat ../../ljspeech/tts1/conf/tuning/train_pytorch_transformer.v1.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TNxNLahYqYFq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cat ../../ljspeech/tts1/conf/tuning/train_fastspeech.v2.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11PdgMfRqYFs",
        "colab_type": "text"
      },
      "source": [
        "We can easily switch the model to be trained by only changing `--train_config`.  \n",
        "(NOTE: FastSpeech needs a teacher model, pretrained Transformer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MlkD_x4wRD80"
      },
      "source": [
        "Let's train the network.  \n",
        "You can specify the config file via `--train_config` option.  \n",
        "It takes several minutes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "cgdY6vp0Bhy4",
        "colab": {}
      },
      "source": [
        "# use modified configuration file as train config\n",
        "!./run.sh --stage 3 --stop_stage 3 --train_config conf/train_pytorch_tacotron2_mini.yaml --verbose 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "HsN8qD1gKnTc"
      },
      "source": [
        "You can see the training log in `exp/train_*/train.log`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "shtiZLczqYFv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cat exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/train.log"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wy0rXqBNTAtk"
      },
      "source": [
        "The models are saved in `exp/train_*/results/` directory.\n",
        "\n",
        "- `exp/train_*/results/model.loss.best`: contains only the model parameters.  \n",
        "- `exp/train_*/results/snapshot.ep.*`: contains the model parameters, optimizer states, and iterator states. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "YRxIFjnWS_t9",
        "colab": {}
      },
      "source": [
        "!tree -L 1 exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/results"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_vyKmHl8T5uj"
      },
      "source": [
        "`exp/train_*/results/*.png` are the figures of training curve.  \n",
        "Let us check them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Dwee4a8NUOBg",
        "colab": {}
      },
      "source": [
        "from IPython.display import Image, display_png\n",
        "print(\"all loss curve\")\n",
        "display_png(Image(\"exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/results/all_loss.png\", width=500))\n",
        "print(\"l1 loss curve\")\n",
        "display_png(Image(\"exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/results/l1_loss.png\", width=500))\n",
        "print(\"mse loss curve\")\n",
        "display_png(Image(\"exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/results/mse_loss.png\", width=500))\n",
        "print(\"bce loss curve\")\n",
        "display_png(Image(\"exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/results/bce_loss.png\", width=500))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ui57IrR4VCpv"
      },
      "source": [
        "`exp/train_*/results/att_ws/*.png` are the figures of attention weights in each epoch.  \n",
        "In the case of E2E-TTS, it is very important to check that they are diagonal."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "NyMW2tx0VKjC",
        "colab": {}
      },
      "source": [
        "print(\"Attention weights of initial epoch\")\n",
        "display_png(Image(\"exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/results/att_ws/fash-cen1-b.ep.1.png\", width=500))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GMfEk7o5qYF5",
        "colab_type": "text"
      },
      "source": [
        "Example of a good diagonal attention weights:\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/ex_attention_weights.png?raw=1 width=60%>\n",
        "</div>\n",
        "We should monitor whether the attention weight becomes like this figure."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JjkrYF-NeZoM"
      },
      "source": [
        "Also, we support tensorboard.  \n",
        "You can see the training log through tensorboard."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "OqwnbGznmrgb",
        "colab": {}
      },
      "source": [
        "# only available in colab\n",
        "%load_ext tensorboard\n",
        "%tensorboard --logdir tensorboard/train_nodev_pytorch_train_pytorch_tacotron2_mini/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "88UvHDgGJSLX"
      },
      "source": [
        "### Stage 4: Network decoding\n",
        "\n",
        "This stage performs decoding with trained model.\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_stage4.png?raw=1 width=80%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ixds4YtjqYF8",
        "colab_type": "text"
      },
      "source": [
        "Decoding parameters can be specified by `--decode_config`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MfShBa1uqYF8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!head -n 32 run.sh | tail -n 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-p7-IKa5qYF-",
        "colab_type": "text"
      },
      "source": [
        "Decoding configuration in written in `.yaml` format file.  \n",
        "Let us check the default configuration `conf/decode.yaml`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tq5RGJZyqYF-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cat conf/decode.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sq2SI4b2qYGA",
        "colab_type": "text"
      },
      "source": [
        "Let us modify to stop the generation in early steps."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B8ZKJNdjqYGA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# load configuration yaml\n",
        "import yaml\n",
        "with open(\"conf/decode.yaml\") as f:\n",
        "    params = yaml.load(f, Loader=yaml.Loader)\n",
        "\n",
        "# change hyperparameters by yourself!\n",
        "params.update({\n",
        "    \"maxlenratio\": 1.0,\n",
        "})\n",
        "\n",
        "# save\n",
        "with open(\"conf/decode_mini.yaml\", \"w\") as f:\n",
        "    yaml.dump(params, f, Dumper=yaml.Dumper)\n",
        "\n",
        "# check modified version\n",
        "!cat conf/decode_mini.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "PjVP2QPXBmav",
        "colab": {}
      },
      "source": [
        "# run stage 4 and then stop\n",
        "!./run.sh --stage 4 --stop_stage 4 --nj 2 --verbose 1 \\\n",
        "    --train_config conf/train_pytorch_tacotron2_mini.yaml \\\n",
        "    --decode_config conf/decode_mini.yaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "I7-8ja9kcdDG"
      },
      "source": [
        "Generated features are saved as `ark/scp` format.  \n",
        "Also figures of attention weights and stop probabilities are saved as `{att_ws/probs}/*.png`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8V_g_VbDckjT",
        "colab": {}
      },
      "source": [
        "!tree -L 2 exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/outputs_model.loss.best_decode*"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9A74Jv8KL_FC"
      },
      "source": [
        "### Stage 5: Waveform synthesis\n",
        "\n",
        "This stage synthesizes waveform with Griffin-Lim.\n",
        "\n",
        "<div align=center>\n",
        "    <img src=https://github.com/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_tts/figs/tts_stage5.png?raw=1 width=80%>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wasoVKklc4eS",
        "colab": {}
      },
      "source": [
        "# run stage 5 and then stop\n",
        "!./run.sh --stage 5 --stop_stage 5 --nj 2 \\\n",
        "    --train_config conf/train_pytorch_tacotron2_mini.yaml \\\n",
        "    --decode_config conf/decode_mini.yaml \\\n",
        "    --griffin_lim_iters 4"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fNpjsAFxqYGG",
        "colab_type": "text"
      },
      "source": [
        "Please run stage 5."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "E7eoOkRYeIhN"
      },
      "source": [
        "Generated wav files are saved in \n",
        "- `exp/train_nodev_pytorch_*/outputs_model.loss.best_*_denorm/{train_dev,test}/wav/`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kp6LcFfih0hJ",
        "colab": {}
      },
      "source": [
        "!tree -L 2 exp/train_nodev_pytorch_train_pytorch_tacotron2_mini/*_denorm"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NvW10my7qYGI",
        "colab_type": "text"
      },
      "source": [
        "Now you finish building your own E2E-TTS model!\n",
        "\n",
        "But unfortunately, this model cannot generate a good speech.  \n",
        "Let us listen to the samples in demo HP to check the quality.  \n",
        "https://espnet.github.io/espnet-tts-sample/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWOG6pGZqYGI",
        "colab_type": "text"
      },
      "source": [
        "## 3. Demonstration of the use of TTS pretrained models\n",
        "\n",
        "We provide pretrained TTS models and these are easy to use with `espnet/utils/synth_wav.sh`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kql9nwxqqYGI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# move on directory\n",
        "os.chdir(\"../../librispeech/asr1\")\n",
        "!pwd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vtV8Vo3eqYGM",
        "colab_type": "text"
      },
      "source": [
        "Let us check the usage of `espnet/utils/synth_wav.sh`.  \n",
        "It will automatically downloads pretrained model from online, you do not need to prepare anything.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rTFtllyjqYGM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!../../../utils/synth_wav.sh --help"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9esdnJYqYGN",
        "colab_type": "text"
      },
      "source": [
        "Let us generate your own text with pretrained models!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2NUOuK-WqYGO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# generate your sentence!\n",
        "!rm -rf decode/example\n",
        "print(\"Please input your favorite sentence!\")\n",
        "text = input()\n",
        "text = text.upper()\n",
        "with open(\"example.txt\", \"w\") as f:\n",
        "    f.write(text + \"\\n\")\n",
        "\n",
        "# you can change here to select the pretrained model\n",
        "!../../../utils/synth_wav.sh --stop_stage 3 --models ljspeech.fastspeech.v1 example.txt\n",
        "# !../../../utils/synth_wav.sh --stop_stage 3 --models ljspeech.tacotron2.v3 example.txt\n",
        "# !../../../utils/synth_wav.sh --stop_stage 3 --models ljspeech.transformer.v1 example.txt\n",
        "\n",
        "# check generated audio\n",
        "from IPython.display import display, Audio, Image, display_png\n",
        "display(Audio(\"decode/example/wav/example_1.wav\"))\n",
        "!sox decode/example/wav/example_1.wav -n rate 22050 spectrogram\n",
        "display_png(Image(\"spectrogram.png\", width=750))\n",
        "\n",
        "# check attention and probs\n",
        "if os.path.exists(\"decode/example/outputs/att_ws/example_1_att_ws.png\"):\n",
        "    display_png(Image(\"decode/example/outputs/att_ws/example_1_att_ws.png\", width=1000))\n",
        "    display_png(Image(\"decode/example/outputs/probs/example_1_prob.png\", width=500))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_kdTdcpcqYGQ",
        "colab_type": "text"
      },
      "source": [
        "Also you can try the neural vocoder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r54NtqwAqYGQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# generate your sentence!\n",
        "!rm -rf decode/example_short\n",
        "print(\"Please input your favorite sentence!\")\n",
        "text = input()\n",
        "text = text.upper()\n",
        "with open(\"example_short.txt\", \"w\") as f:\n",
        "    f.write(text + \"\\n\")\n",
        "    \n",
        "# extend stop_stage\n",
        "!../../../utils/synth_wav.sh --stop_stage 4 --models ljspeech.tacotron2.v3 example_short.txt\n",
        "\n",
        "# check generated audio\n",
        "display(Audio(\"decode/example_short/wav/example_short_1.wav\"))\n",
        "display(Audio(\"decode/example_short/wav_wnv/example_short_1_gen.wav\"))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xJjZ15bgqYGS",
        "colab_type": "text"
      },
      "source": [
        "## 4. Demonstration of the use of ASR pretrained models\n",
        "\n",
        "ESPnet also provides the `espnet/utils/recog_wav.sh` to use pretrained ASR models.  \n",
        "Let us recognize the generated speech!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jcAl-on8qYGT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!../../../utils/recog_wav.sh --help"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8W5zof2qYGU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# downsample to 16 kHz for ASR model\n",
        "!sox decode/example/wav/example_1.wav -b 16 decode/example/wav/example_1_16k.wav rate 16k pad 0.1 pad 0 0.1\n",
        "\n",
        "# make decode config\n",
        "import yaml\n",
        "with open(\"conf/decode_sample.yaml\", \"w\") as f:\n",
        "    yaml.dump({\n",
        "        \"batchsize\": 0,\n",
        "        \"beam-size\": 5,\n",
        "        \"ctc-weight\": 0.4,\n",
        "        \"lm-weight\": 0.6,\n",
        "        \"maxlenratio\": 0.0,\n",
        "        \"minlenratio\": 0.0,\n",
        "        \"penalty\": 0.0,\n",
        "    }, f, Dumper=yaml.Dumper)\n",
        "\n",
        "# let's recognize generated speech\n",
        "!../../../utils/recog_wav.sh --models librispeech.transformer.v1 \\\n",
        "    --decode_config conf/decode_sample.yaml \\\n",
        "    decode/example/wav/example_1_16k.wav"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vx0xDzK0qYGW",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "- Can build E2E-TTS models with unified-design recipe\n",
        "- Can try various models by just changing the yaml file\n",
        "\n",
        "Through ESPnet, you can build / use E2E-TTS and E2E-ASR in the same manner!\n",
        "\n",
        "Thank you for your attention!\n",
        "\n",
        "[*Go to the next notebook from here!*](https://colab.research.google.com/github/espnet/interspeech2019-tutorial/blob/master/notebooks/interspeech2019_asr/interspeech2019_asr.ipynb)"
      ]
    }
  ]
}
